Field of the Invention
The present invention relates to an image processing apparatus, an image processing method, and a storage medium.
Related Art
In recent years, an image capturing apparatus that introduces information for direction distribution of incident light rays, i.e. an image capturing apparatus called a “plenoptic camera”, has been developed (for example, Japanese Unexamined Patent Application (Translation of PCT Publication), Publication No. 2009-532993.
In a plenoptic camera, a microscopic lens array is arranged which is configured by arranging a plurality of micro lenses (hereinafter referred to as “micro-lens”) in a grid pattern between a main lens, which is a conventional imaging lens, and the imaging element.
The individual micro-lenses constituting the micro-lens array condense light condensed by the main lens onto a plurality of pixels in the imaging element, depending on the angle at the light ray arrives. The plenoptic camera generates a captured image (hereinafter, referred to as a “light field image”) by combining an image condensed onto each of the pixels in the imaging element by the individual micro-lenses (hereinafter, referred to as “sub-image”).
The light field image is generated by light irradiated through the conventional main lens, as well as the micro-lenses array in this way. In other words, the light field image not only has spatial information of two-dimensions that is included in a conventional captured image, but as information not included in conventionally captured images, also has directional information of two-dimensions indicating what direction the arriving light rays are from when viewing from the imaging element.
After capturing a light field image by employing such two-dimensional directional information, the plenoptic camera, using the data of this light field image, can reconstruct the figure on a plane separated in front by an arbitrary distance during image capture. In other words, even in a case of having captured a light field image without the plenoptic camera focusing at a predetermined distance, after image capturing, the plenoptic camera can freely produce data of an image (hereinafter referred to as a “reconstructed image”) equivalent to an image captured by focusing at this predetermined distance using the data of this light field image.
More specifically, the plenoptic camera sets one point on a plane separated by an arbitrary distance to an attention point, and calculates on which pixels in the imaging element the light from this attention point is distributed via the main lens and the micro-lens array.
In this regard, if each pixel of the imaging element corresponds to each pixel constituting the light field image, for example, then the plenoptic camera integrates the pixel value of at least one pixel on which light from the attention point is distributed, among each of the pixels constituting the light field image. This value thus calculated becomes the pixel value of the pixel corresponding to the attention point in the reconstructed image. The pixel corresponding to the attention point in the reconstructed image is thereby reconstructed.
The plenoptic camera sequentially sets each pixel corresponding to each point on a plane separated by an arbitrary distance from the plenoptic camera (each pixel constituting the reconstructed image) to the attention point, respectively, and repeats the aforementioned sequence of processing, thereby reconstructing data of the reconstructed image (aggregate of pixel values of each pixel of the reconstructed image).
Incidentally, as shown in FIG. 14, a conventional plenoptic camera constitutes a micro lens array with one kind of the micro lens and covers the entire focusing area by the one kind of the micro lens. Therefore, blurring through the micro lens (micro lens blurring) may develop depending on a distance to a subject and a value of a focusing distance of the micro lens, which becomes a hindrance upon generating a reconstructed image of high definition using a light field image captured.
Furthermore, in a case of reducing the thickness of a plenoptic camera, it is necessary to reduce the distance between the main lens and the micro lens array. However, in a case of reducing the distance between the main lens and the micro lens array, the change of magnitude of the micro lens blurring corresponding to the change of the distance to a subject becomes greater. For example, in a case of the distance between the main lens and a subject being infinity, although the micro lens blurring is minimum, the magnitude of the blurring through the micro lens becomes greater as the distance therebetween approaches from infinity.